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ABSTRACT 

Recent  years  have  seen  huge  increases  both  in  computing  power  and  the  number  of  people  able  to  access 
computers  and  the  Internet.  This  proliferation  of  information  and  communication  technologies  has 
enabled  higher  quality  learning  to  be  made  available  through  increasingly  sophisticated  modes  of 
presentation.  Traditional  or  conventional  training  programs  use  a  variety  of  instructional  development 
strategies  to  support  a  student's  need  to  master  a  variety  of  competencies.  Simulations  and  games  are 
increasingly  being  deployed  as  powerful  and  valuable  extensions  to  these  traditional  educational 
initiatives. 

However,  learning  is  a  comprehensive  process  which  does  not  simply  consist  of  the  transmission  and 
learning  of  content.  While  simulations  offer  the  opportunity  to  undergo  informative  interactive 
experiences,  they  do  not,  by  themselves,  constitute  training  or  instruction.  Assessment,  student  tracking 
and  feedback  are  important  elements  in  the  teaching  and  learning  process. 

Recognizing  the  importance  of  these  requirements,  two  IEEE  standards  committees  have  formed  a 
collaborative  study  group  to  investigate  the  potential  of  formalizing  a  standard  set  of  technical 
specifications  to  allow  simulations  and/or  games  to  be  launched  and  managed  through  SCORM- 
conformant  content  and  Learning  Management  Systems. 

This  paper  and  presentation  will  focus  on  discussions,  both  technical  and  pedagogical;  to  address  the 
many  issues  associated  with  developing  such  SCORM-Simulation  Interface  standards.  Discussions  will 
focus  on  the  different  use  cases  for  simulations  and  the  key  interface  points  between  simulation  content 
and  LMS  environments  such  as  delivering  simulation  content  to  the  learner,  monitoring  key  interactions 
and  performance  within  simulation  content  and  determining  what  the  student  should  next  experience 
within  the  continuum  of  training. 

Additional  information  may  also  be  found  at  the  SCORM-Sim  Interface  standards  study  group  section  of 
the  SISO  website  at  http://www.sisostds.org 
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PREFACE 

The  technologies  used  for  Modeling,  Simulation  and  gaming  have  great  potential  to  become  powerful  and 
valuable  extensions  to  traditional  educational  initiatives.  Cognitive  skills  can  be  practiced  and  honed  as 
students  interact  with  models,  simulations  and  virtual  environments  and  research  shows  that  people 
perform  better  after  instruction  when  they  have  learned  in  the  context  of  doing  [1].  However,  these 
technologies  are  only  part  of  the  mix  of  learning  strategies  that  must  be  experienced  by  the  learner  in  order 
to  create  a  well  educated  individual. 

Student  tracking  and  assessment  are  important  elements  in  the  learning  process  and  play  a  key  role  in 
determining  when  the  student  is  ready  to  proceed  to  a  more  challenging  level  of  training  [2,3,4].  The 
combination  of  using  advanced  modeling,  simulation  and  gaming  techniques  with  the  advanced  learner 
tracking  capabilities  of  a  Learning  Management  System  (LMS)  has  great  potential  to  increase  the  efficacy 
of  these  technologies  while  reducing  costs  and  optimizing  performance. 

This  paper  will  discuss  the  progress  of  a  collaborative  IEEE/SISO  study  group  assigned  to  study  the 
underlying  technology  and  methods  of  using  modeling,  simulation  and  gaming  technologies  within  a 
SCORM-managed  learning  environment.  The  study  group  has  solicited  and  collected  position  papers 
from  Industry,  Academia  and  Government.  This  call  was  met  with  an  International  response.  A  primary 
focus  of  the  group  is  to  compile  these  papers  to  document  potential  use  cases,  identify  common 
architectural  components  and  recommend  any  potential  standards  necessary  to  facilitate  this  integration. 
This  paper  represents  a  status  report  on  this  effort  and  will  summarize  the  group’s  initial  review  of  these 
position  papers.  It  also  discusses  potential  areas  for  standards  development,  the  future  focus  of  this  group 
and  an  associated  roadmap  for  how  this  integration  may  be  accomplished. 


OVERVIEW 

The  delivery  of  web-deployable  training  content  has  been  standardized  under  the  Advanced  Distributed 
Learning  (ADL)  Initiative  and  is  becoming  very  popular  and  widespread  throughout  the  world.  A  key 
element  of  this  effort  is  the  SCORM  specification,  a  set  of  standards  to  define  the  interface  between  a 
Learning  Management  System  and  learning  content. 

Within  the  SCORM  context,  the  term  LMS  implies  a  web  server-based  environment  in  which  the 
intelligence  resides  for  controlling  the  delivery  of  learning  content  to  students.  This  involves  managing 
student  information,  delivering  content,  monitoring  key  interactions  and  performance  within  the  content 
and  then  determining  what  the  student  should  next  experience.  In  the  future,  instructional  strategies  will 
include  a  range  of  training  systems,  simulations  and  devices  in  a  range  of  different  use  cases. 

The  purpose  of  SCORM  is  to  provide  a  means  for  interoperability  between  learning  content,  in  the  form  of 
Sharable  Content  Objects  (SCOs)  and  LMSs.  For  this  to  be  possible,  there  must  be  a  common  way  to 
launch  content,  a  common  way  for  content  to  communicate  with  an  LMS  and  predefined  data  elements 
that  are  exchanged  between  an  LMS  and  content  during  its  execution.  As  presently  defined,  a  SCO  can  be 
an  entire  course,  a  lesson  within  a  course,  or  a  topic  within  a  lesson.  SCORM  is  presently  focused  on 
packaging  these  objects  to  enable  a  “Learning  Object”  approach  to  training.  In  SCORM,  learning  content 
can  be  described,  sequenced,  tracked  and  delivered. 

When  evaluating  games  and  simulations  as  a  form  of  learning  content,  we  will  often  find  more  complex 
requirements  for  delivering,  initializing,  launching  tracking  and  management  than  what  is  currently 
possible  within  SCORM.  Complicating  this  further,  are  the  many  types  of  simulations  and  many  methods 
for  how  they  are  used. 
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The  SCORM  Content  Aggregation  Model 

Empirical  studies  such  as  those  summarized  in  Perrin  et  al.  [5]  and  described  in  Perrin  et  al.  [3,4]  have 
shown  that  an  optimum  learning  approach  is  to  create  learning  strategies  that  conform  to  the  evolving  skill 
level  of  the  student  and  provide  various  types  of  feedback  on  performance.  In  the  SCORM  2004 
environment,  content  objects  do  not  determine,  by  themselves,  how  to  traverse  through  a  unit  of 
instruction.  Instead,  the  LMS  processes  “Sequencing  and  Navigation”  (S&N)  rules  using  results  from  the 
content  objects  to  determine  the  order  in  which  a  student  will  experience  learning  resources. 

The  SCORM  Content  Aggregation  Model  (CAM)  describes  the  methods  used  to  assemble  and  sequence 
learning  resources  for  the  purpose  of  delivering  a  desired  learning  experience.  A  learning  resource  is  any 
representation  of  information  that  is  used  in  this  learning  experience.  In  order  to  use  interactive  models, 
simulations  and  games  as  learning  resources,  a  method  of  integrating  them  into  the  SCORM  paradigm  is 
necessary.  A  key  enabler  being  looked  at  as  one  potential  area  for  standards  development  is  a  mechanism 
of  describing,  launching  and  initializing  these  simulation  resources  from  content  objects  or  directly  by  the 
LMS. 

The  SCORM  Run-Time  Environment 

The  SCORM  Run-Time  Environment  (RTE)  Document  specifies  the  launch  of  learning  content, 
communication  between  content  and  an  LMS,  data  transfer,  and  error  handling.  When  the  focus  is  shifted 
to  interactive  simulations  specifically,  the  issues  involved  are  greatly  amplified.  There  are  many  types  of 
simulations  with  varying  levels  of  interactivity  and  complexity.  Simulations  range  from  full  motion  or 
fully  immersive  simulators  and  multiplayer  distributed  games  to  simple  user  interface  “surface 
simulations.”  They  can  be  resident  in  a  client  browser  or  embedded  into  a  separate  system.  Adding  to  the 
difficulty  is  the  fact  that  often  a  simulation’s  resources  require  auxiliary  processes  and  systems  that  must 
be  initialized,  configured  and  available  for  a  simulation-based  application  to  launch. 

The  LMS  in  the  SCORM  architecture  is  required  to  operate  in  a  web-based  server/browser  infrastructure 
to  deliver  and  receive  communications  from  web  pages.  Therefore,  the  LMS  has  no  inherent  capabilities 
to  directly  launch  or  communicate  with  a  native  application  session  running  on  the  client  or  other  machine. 
One  common  approach  presented  to  the  study  group  was  to  use  a  Java  applet  or  ActiveX  component 
embedded  in  an  HTML  web  page  of  the  learning  content  to  launch  a  simulation. 

Many  of  these  approaches  also  used  an  applet  or  component  to  create  and  manage  a  message  pipeline 
between  the  simulation  and  the  LMS.  In  the  SCORM  environment,  the  LMS  is  responsible  for 
establishing  a  “handshake”  with  the  client  browser  for  the  exchange  of  data  during  each  learning  session. 
Communications  from  the  content  to  the  LMS  are  accomplished  using  ECMAScript,  which  can  be  used  to 
“set”  and  “get”  data  values  on  the  LMS  server  using  the  SCORM-conformant  Application  Programming 
Interface  (API)  provided  by  the  LMS. 

The  SCORM  RTE  defines  a  specific  data  model,  which  is  the  required  vocabulary  that  must  be  used  when 
communicating  with  the  LMS.  These  “assessment  modules”  collect  messages  from  the  simulations  which 
are  translated  into  assessment  information  that  an  LMS  can  understand.  This  process  was  implemented 
differently  across  the  position  papers.  Many  approaches  used  an  HLA  federate  to  allow  the  assessment 
module  to  subscribe  to  published  events  from  the  simulation.  Others  used  a  similar  approach  without 
using  HLA. 
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Joint  ADL  Co-Laboratory’s  Integrated  Prototype  Architecture  -  2012 

While  the  approach  of  using  a  Java  applet  proves  adequate  in  many  cases,  there  were  concerns  expressed 
from  the  Military  members  of  the  study  group  about  the  security  considerations  of  using  Mobile  Code  to 
launch,  initiate  and  communicate  between  the  simulation  and  the  LMS.  With  the  introduction  of  the  2012 
-  Integrated  Prototype  Architecture  (IP A),  the  Joint  ADL  Co-Laboratory  has  introduced  a  new  SCORM 
object  called  a  “Lightweight  Scenario  Format  (LSF)”  file.  The  Co-Lab  is  careful  to  mention  that  the  LSF 
is  not  intended  to  be  a  common  configuration  file  format  with  the  ability  to  initialize  a  simulation.  Rather, 
the  LSF  file  specifies  how  a  particular  simulation  should  be  configured  to  support  the  objectives  of  a 
course. 

LSF  files  are  represented  using  an  XML-based  syntax  and  contain  high-level  information  that  can  describe 
key  elements  of  a  simulation  scenario.  These  elements  are  similar  to  those  which  are  described  by  the 
DARWARS  OCM  model  -  Objectives,  Conditions,  and  Measures. 

•  Objectives  can  be  correlated  with  IMS  Simple  Sequencing  objectives. 

•  Conditions  can  describe  start-up  factors  like  equipment  types,  locations,  environmental  factors, 
etc. 

•  Measures  describe  assessable  factors  which  can  be  used  to  provide  values  to  objectives. 

The  IPA-2012  is  built  around  the  concept  of  a  Distributed  Training  Event  Coordinator  (DTEC)  that  works 
with  an  LMS  to  assist  users  in  the  process  of  locating  and  initializing  the  best,  most  accessible  training 
system  that  is  available  to  them  at  any  given  moment.  This  process  starts  with  embedding  an  LSF  file 
within  a  piece  of  courseware. 

The  DTEC  follows  closely  the  work  of  the  DARWARS  initiative  and  is  envisioned  as  a  companion  to 
existing  LMSs,  not  a  replacement.  Rather  than  attempting  to  embed  a  simulation  within  a  SCO,  the  SCO 
might  refer  instead  to  a  set  of  objectives,  conditions,  measures  or  a  specific  simulation,  and  delegate 
responsibility  for  managing  this  element  to  a  local  instance  of  the  DARWARS  Core.  This  linkage  would 
mediate  the  exchange  of  information  between  the  LMS  and  identify  simulation  scenarios  that  are 
appropriate  for  the  unfolding  learning  event. 

When  the  user  is  ready  to  begin  executing  a  simulation,  the  LMS  initiates  a  DTEC-provided  “lobby.”  This 
is  essentially  a  web  page  which  offers  a  brief  idle  period  during  which  simulation  participants  can  gather. 
The  lobby  provides  learners  with  a  smooth  transition  from  an  asynchronous  to  a  synchronous  learning 
environment.  When  all  roles  have  been  filled,  the  DTEC  initializes  the  simulation  and  waits  for  measures 
to  be  reported.  These  measures  are  passed  back  to  the  originating  LMS  where  they  are  used  to  assign 
values  to  Simple  Sequencing  Objectives. 


COMMUNICATIONS  AND  DATA  MODELS 

Security  implications  for  web  browser/server  architectures  limit  how  an  LMS  interacts  with  learning 
content;  consequently,  the  LMS  is  not  able  to  “watch”  what  is  happening  inside  a  simulation  and  take 
immediate  action. 

Figure  1  shows  key  elements  and  interfaces  that  have  emerged  from  the  study  group  discussions.  This 
diagram  is  color  coded.  Yellow  is  used  to  represent  simulation-related  information. 

A  stand-alone  simulation  takes  initial  conditions  and  produces  events. 
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In  the  SCORM  context,  the  Learning  Management  System  (shown  as  the  blue  oval)  is  responsible  for 
managing  three  forms  of  data  (shown  in  medium  blue): 

•  Learner  data,  including  learner  identification  and  learner  assessment  records. 

•  Competency  information,  shown  here  in  terms  of  tasks,  conditions,  and  standards,  but  possibly 
including  other  forms,  such  as  learning  objectives. 

•  Content,  the  traditional  SCOs,  typically  HTML-based  data. 


Figure  1:  Overview  of  LMS  and  Simulation  Interfaces 


The  new  piece  is  the  assessment  engine,  which  takes  Assessment  Rules  (an  algorithmic  form  of  policy) 
and  streams  of  Events  generated  by  the  Simulation,  and  generates  Assessment  Results. 

The  arrows  shown  in  this  diagram  are  natural  locations  for  interface  specifications.  The  Distributed 
Interactive  Simulation  (DIS)  and  HLA  standards  already  provide  standard  data  models  for  event  streams. 
A  couple  of  other  SISO  groups  are  working  on  standards  for  Initial  Conditions.  SCORM  2004  provides 
standard  specifications  for  Assessment  Results  and  rollup  functions  to  aggregate  assessment  results. 
However,  the  least  well  understood  interface  is  the  process  of  linking  Competency  Data  to  Assessment 
Rules  and  Initial  Conditions.  Furthermore,  there  is  a  need  for  managing  the  interaction  between  these 
subsystems  and  for  configuring  a  learning  system  by  distributing  these  functions  over  multiple  resources 
in  the  context  of  restrictive  computer  networks. 

In  order  for  a  SCORM-based  LMS  to  track  student  performance  during  a  simulation,  the  data 
communicated  to  the  LMS  must  be  confined  to  the  data  model  that  SCORM  provides.  Dodds  &  Fletcher 
[6]  state  that  “ SCORM  presently  provides  a  rules-based  ’ learning  strategy’  that  enables  Sharable  Content 
Objects  (SCOs)  to  set  the  state  of  global  records  called  objectives.  These  records  can  store  the  learner’s 

degree  of  mastery  in  the  form  of  a  score  or  a  pass/fail  state .  A  'hook’  was  included  in  the  record  that 

permits  them  to  reference  externally  defined  competencies.  As  the  learner  is  sequenced  through  the  SCOs , 
the  learning  system  builds  up  a  representation  of  the  learner’s  mastery  and  progress.  The  objective 
records  may  be  viewed  as  a  simple  model  of  the  learner’s  state  (Gibbons  &  Fairweather,  799<§).”[6,7] 
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Generally  speaking,  a  simulation  will  typically  contain  a  number  of  “objectives”,  and  learner  progress 
toward  each  objective  is  tracked.  To  do  this,  a  simulation  has  the  potential  to  generate  massive  amounts  of 
data  that  characterize  the  complex  relationships  between  interactions  for  each  objective.  A  method  of 
hierarchically  organizing  simulation  events,  task  steps,  supporting  skills  and  training  objectives  needs  to 
be  articulated  such  that  a  student’s  actions  can  be  compared  with  a  rule-based  model  of  assessment. 

While  a  simulation  may  track  many  assessment  variables  internally,  it  needs  to  be  able  to  combine  these 
variables  into  data  values  that  an  LMS  is  able  to  understand.  This  appears  to  be  a  common  consideration 
across  the  study  group  and  has  been  identified  as  an  area  that  warrants  further  investigation  for  the 
development  of  potential  standards.  As  shown  in  the  above  figure,  there  is  a  mechanism  to  listen  to  events 
or  messages  from  a  simulation,  process  them,  translate  them  into  data  that  an  LMS  can  understand  and 
communicate  them  to  the  SCORM  data  model  either  through  the  LMS  or  via  web  services. 


OTHER  APPROACHES  &  INITIATIVES 

As  the  assessment  component  processes  events  from  the  simulation,  there  is  potential  to  collect  other 
contextual  information  that  can  be  used  as  evidence  to  support  competency.  From  an  instructional 
effectiveness  point  of  view,  the  capture  of  this  information  could  greatly  increase  the  effectiveness  of 
simulations  by  allowing  a  more  in-depth  review  of  what  a  student  has  and  hasn’t  mastered. 

Dodds  &  Fletcher  [6]  further  state  that  “Another  emerging  specification  called  IMS  Reusable  Definition  of 
Competency  or  Educational  Objective  (2002)  defines  a  means  of  building  a  taxonomy  of  competency 
definitions  that  meet  specific  objectives.  This  taxonomy  may  be  organized  hierarchically  to  represent 
dependencies,  supporting  skills,  or  prerequisites.  Each  competency  definition  has  a  text  description  of  the 
competency  and  a  unique  identifier  that  may  be  referenced  externally.  The  organization  of  a  competency 
definition  could  represent  specific  skills  or  knowledge  to  be  acquired  for  a  specific  task  or  subject  domain 
(e.g.,  as  one  might  find  in  Quantitative  Domain  Mapping).  Since  objectives  records  in  SCORM  can 
reference  the  Competency  model  identifiers,  the  means  to  compare  the  state  of  the  learner  and  the  desired 
competencies  now  exists.  This  capability  provides  a  system-based  means  to  perform  skills  gap  analysis 
leading  to  more  sophisticated  and  adaptive  strategies  that  use  such  information  (Wiley,  2000).  ” 


STANDARDS  AND  SPECIFICATIONS 
IEEE  LTSC 

The  IEEE  Learning  Technology  Standards  Committee  (LTSC)  (http://ieeeltsc.org  )  is  chartered  by  the 
IEEE  Computer  Society  Standards  Activity  Board  to  develop  accredited  technical  standards, 
recommended  practices,  and  guides  for  learning  technology. 

Active  LTSC  standards  and  standards  projects  include  standards  for: 

•  Learning  Object  Metadata 

•  Communication  between  Learning  Management  Systems  (LMS)  and  content 

•  Encapsulating  definitions  of  competencies 

•  Practices  concerning  digital  rights  expression  languages 

•  Content  aggregations 

•  Simulation  Interface  Standards 
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Currently,  three  of  the  five  components  of  SCORM  2004  (Metadata,  Communications,  and  Content 
Aggregation)  are  based  on  LTSC  standards,  while  a  fourth  (Competency)  is  in  progress. 

The  LTSC  coordinates  formally  and  informally  with  other  organizations  that  produce  specifications  and 
standards  for  similar  purposes.  Standards  development  is  done  in  working  groups  via  a  combination  of 
face-to-face  meetings,  teleconferences,  and  exchanges  on  discussion  groups.  The  LTSC  is  a  Category  C 
Liaison  Organization  with  ISO/IEC  (JTC  1)  Subcommittee  36  (SC36)  for  the  development  of  learning, 
education,  and  technology  standards.  The  LTSC  is  governed  by  a  Sponsor  Executive  Committee  (SEC) 
consisting  of  working  group  chairs  and  elected  officers  as  described  at  http://ieeeltsc.org/. 

The  IEEE  LTSC  is  currently  going  through  a  balloting  process  on  a  specification  for  “Standard  for 
Reusable  Competency  Definitions”  [8].  This  standard  is  motivated  in  part  by  a  growing  international 
movement  (led  by  the  Human  Resources  community)  to  look  at  the  bigger  picture  of  expressing  objectives 
and  relationships  among  them.  At  issue  is  how  to  express  the  fact  that  one  objective  (or  competency) 
might  be  “composed”  of  several  sub-objectives  and  how  to  express  ways  in  which  data  on  sub-objectives 
can  be  “rolled  up”  [9].  There  is  also  a  focus  on  being  able  to  reference  existing  taxonomies  to  create 
competency  definitions. 

The  “Computer-Managed  Instruction”  (CMI)  working  group  within  the  LTSC  has  been  actively  pursuing 
standards  related  to  SCORM,  including  one  that  is  defining  a  nomenclature  and  a  conceptual  model  for 
digital  aggregates  of  resources  for  learning,  education,  and  training  applications.  The  motivation  for  this 
is  that  different  domains  (multimedia,  libraries,  learning  technology,  technical  documentation,  etc.)  have 
all  created  standards  for  “packaging”  digital  resources,  metadata  about  the  resources,  instructions  for  using 
the  resources,  and  other  information.  The  DARWARS  Training  Package  is  a  good  example.  The  LTSC 
activity,  known  as  RAMLET,  is  an  attempt  to  help  training  systems  make  use  of  content  that  is  packaged 
in  different  formats  and  to  exchange  training  content  with  systems  from  other  domains. 

SISO 

The  Simulation  Interoperability  Standards  Organization  (SISO)  originated  over  ten  years  ago  with  a  small 
conference  held  April  26  and  27,  1989,  called,  "Interactive  Networked  Simulation  for  Training."  The 
original  conference  attracted  approximately  60  people.  The  group  was  concerned  that  there  was  activity 
occurring  in  networked  simulation,  but  that  it  was  occurring  in  isolation.  The  group  believed  that  if  there 
were  a  means  to  exchange  information  between  companies  and  groups  that  the  technology  would  advance 
more  rapidly.  The  group  also  believed  that  once  the  technology  begins  to  stabilize  then  there  would  also 
be  a  need  for  standardization.  The  technology  and  the  consensus  of  the  community  would  be  captured  in 
the  standards  as  networking  or  simulation  technology  matured. 

SISO  is  (now)  an  international  organization  dedicated  to  the  promotion  of  modeling  and  simulation 
interoperability  and  reuse  for  the  benefit  of  a  broad  range  of  M&S  communities  [10].  SISO's  Conference 
Committee  organizes  Simulation  Interoperability  Workshops  (SIWs)  in  the  US  and  Europe.  SISO's 
Standards  Activity  Committee  develops  and  supports  simulation  interoperability  standards,  both 
independently  and  in  conjunction  with  other  organizations.  SISO’s  Standards  Activity  Committee  (SAC) 
is  recognized  as  a  Standards  Development  Organization  (SDO)  by  NATO  and  as  a  Standards  Sponsor  by 
IEEE  responsible  for  the  development  and  maintenance  of  the  IEEE  1516  series,  High  Level  Architecture 
(HLA),  and  IEEE  1278  services,  Distributed  Interactive  Simulation  (DIS).  In  addition,  SISO  is  a 
Category  C  Liaison  Organization  with  ISO/IEC  (JTC  1)  for  the  development  of  standards  for  the 
representation  and  interchange  of  data  regarding  Synthetic  Environment  Data  Representation  and 
Interchange  Specification  (SEDRIS).  Details  about  SISO’s  activities  and  opportunities  to  participate  can 
be  found  on  the  SISO  web  site,  http://www.sisostds.org. 
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i  Activity 


Figure  2.  SISO  Standards  Development  Process 


HLA  and  DIS  are  particularly  germane  to  this  effort  because  of  the  focus  on  standards-based 
interoperability.  HLA  and  DIS  are  the  predominant  simulation  interoperability  standards  within  military 
simulation.  As  such,  any  standard  for  SCORM-Sim  interoperability  although  required  to  be  independent 
of  any  single  simulation  interoperability  standard,  must  work  with  these  two  extant  standards  at  a 
minimum  [11]. 

The  effort  described  in  this  paper  is  being  performed  under  the  auspices  of  the  SISO  SAC.  The  process  by 
which  standards  are  developed  under  the  guidance  of  the  SAC  is  illustrated  in  Figure  2.  Notice  that  the 
input  that  starts  this  process  is  a  product  nomination.  A  product  nomination  is  created  when  a  group  of 
dedicated  proponents  determines  that  there  is  sufficient  interest  and  technical  grounding  to  produce  a 
standard.  In  many  cases,  that  determination  is  made  through  the  efforts  of  a  Study  Group  (SG)  made  up  of 
such  dedicated  proponents  who  research  and  brainstorm  the  desirability  and  feasibility  of  a  standard  in  a 
particular  domain  of  simulation  interoperability.  This  paper  describes  the  efforts  of  such  a  group 
comprised  of  proponents  with  individual,  but  overlapping,  experiences  integrating  simulations  with 
SCORM-conformant  instructional  content. 


Terms  of  Reference 

When  a  SISO  SG  is  established,  the  proponents  must  produce  a  Terms  of  Reference  that  describes  the 
tasks  and  deliverables  for  the  SG.  The  SCORM-Sim  SG  will  execute  the  following  tasks: 

1 .  Call  for  position  papers 

2.  Survey  technical  and  pedagogical  approaches  taken  to  date 

3.  Determine  where  there  is  common  ground 

4.  Discuss  potential  standardization  efforts 

5.  Produce  appropriate  SISO  Product  Nominations  (PNs)  and/or  Product  Authorization  Requests 
(PARs)  for  IEEE  standards  projects 
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Products  resulting  from  the  establishment  and  execution  of  the  SCORM-Sim  SG  will  include: 

1 .  Compilation  of  position  papers 

2.  Identification  of  potential  standards 

3.  Multi-part  PN/PAR  for  a  set  of  standards  whose  development  is  seen  as  valuable  and  feasible 

4.  Final  Report  (the  intent  is  to  then  turn  this  into  a  SISO  Product  Development  Group(  PDG) 
and/or  IEEE  Working  Group) 

USE  CASES 

Many  of  the  position  papers  submitted  to  the  group  contain  specific  use  cases  describing  previous  and 
current  efforts.  In  order  to  ensure  we  cover  aspects  of  SCORM/Simulation  integration  that  were  not 
addressed  in  the  papers,  we  developed  generic  use  cases.  The  generic  use  cases  were  developed  on  two 
levels.  The  Basic  level  simply  indicated  that  simulations  can  be  “used”  by  SCORM  content  in  three  ways: 

1 .  Simulation  is  used  to  demonstrate  (Show  Me) 

2.  Simulation  is  used  for  experiential  learning  (Let  Me  Try) 

3.  Simulation  is  used  for  testing  (Test  Me) 

More  specific  generic  instructional  uses  cases  can  be  found  in  the  group’s  document  area  on  the  SISO  web 
site  [12].  This  document  includes  sixteen  different  ways  in  which  the  three  basic  cases  can  be 
implemented.  The  document  also  includes  three  or  four  uses  cases  for  each  of  the  users:  Instructor; 
Instructional  Designer  (ID);  Simulation  Developer;  Subject  Matter  Expert  (SME);  Content  Developer; 
Simulation/LMS;  Local  Administrator(s);  and  Learner  Using  Simulation  to  Practice  without  Content. 


TAXONOMY 

Different  communities  of  practice  have  very  different  ideas  of  what  constitutes  a  simulation  and  how 
simulations  should  be  incorporated  into  learning  systems.  Part  of  the  effort  of  this  study  group  is  to 
develop  a  taxonomy  of  learning  systems  that  incorporate  simulations,  often  as  part  of  a  blended  learning 
approach  [13].  This  taxonomy  will  help  to  determine  what  standards  efforts  for  interfaces  between  a 
learning  management  system  and  a  simulation  will  have  the  most  impact.  A  taxonomy  of  these 
simulation-based  learning  systems  may  also  help  a  potential  learning  system  customer  decide  which 
standards  are  appropriate  for  the  desired  application. 

The  taxonomy  characterizes  training  systems  by  a  set  of  attributes,  where  each  attribute  has  a  finite  set  of 
possible  values.  These  attributes  are  chosen  to  inform  the  standards  required  to  implement  the  simulation 
and  to  interface  the  simulation  to  an  associated  learning  environment.  The  taxonomy  is  represented  by  a 
decision  tree  where  all  the  learning  systems  in  a  particular  branch  of  the  decision  tree  have  the  same 
attribute  value.  Thus  selecting  the  values  for  a  set  of  attributes  determines  a  collection  of  possible 
training  system  configurations  and  the  corresponding  interface  requirements.  The  attributes  are  grouped 
around  three  major  themes: 

•  Learning  System  Functionality  Attributes:  These  include  individual  or  collective  learners, 
interactive  or  display  results  simulation,  support  for  instructor  interactions  with  the  learner  and/or 
the  simulation,  and  level  of  assessment  automation. 
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Human  Computer  Interface  Attributes:  These  attributes  describe  requirements  for  learner  and 
instructor  control  of  the  initial  conditions  for  the  simulation  and  for  learner  and  instructor 
temporal  control  during  the  simulation. 
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•  Operational  Environment  Attributes:  These  attributes  describe  requirements  for  how  the  learning 
system  is  configured  in  the  learner’s  environment,  particularly  focusing  on  whether  the  learning 
system  is  distributed  across  a  network  or  local  to  the  learner  and  where  the  instructor  (if  any)  is 
located. 

REQUIREMENTS 

Many  of  the  position  papers  included  requirements.  A  few  of  those  requirements  are  summarized: 

Stand-Alone  Training 

Customers  require  simulations  that  can  be  downloaded  from  the  web  and  run  stand-alone  without 
concurrent  access  to  the  web  as  asynchronous  training.  However,  they  also  want  centralized  student 
record-keeping.  In  this  rapidly  changing  environment,  they  also  want  the  training  and  simulations  to  be 
kept  up-to-date. 

Graduated  Student  Help 

Customers  want  learning  systems  that  provide  various  levels  of  student  help.  These  levels  are 
characterized  in  the  Use  Cases  in  terms  of  Show  Me  (Familiarization),  Let  Me  Try  (Acquire  and  Practice), 
Test  Me  (Validation),  all  of  which  require  different  levels  of  learner  support  [14,15]. 

Performance  Assessment 

Customers  require  learning  systems  that  collect  performance  data  based  on  student  actions  that  can  be 
uploaded  to  the  web  as  evidence  of  competency.  The  performance  data  definitions  (e.g.,  learning 
objectives)  should  be  the  same  for  live  and  simulation-based  training.  In  particular,  these  requirements  are 
defined  by  critical  tasks  needed  within  the  simulation  and  by  performance  measures  defined  for  these 
critical  tasks. 

Concurrent  Assessment  of  Multiple  Skills 

Customers  require  simulations  that  provide  the  ability  for  students  to  demonstrate  competency  and  the  ability 
to  integrate  skills  into  performance  on  complex  tasks.  A  single  simulation  session  can  provide  feedback  on 
multiple  learning  objectives  [3,16,17,18,19].  The  SCORM  2004  sequencing  and  navigation  rules  provide 
the  methods  for  rolling  up  data  on  multiple  learning  objectives  to  provide  an  overall  competency  rating. 
However,  it  may  take  several  simulation  sessions  to  obtain  all  the  data  needed  for  an  overall  competency 
rating  [5].  This  may  require  use  of  global  objectives,  which  SCORM  2004  provides  [17,4]. 

Bookmarking 

Customers  require  simulations  that  can  be  bookmarked  to  interrupt  a  session  and  then  resume  the  session 
later.  A  key  issue  is  where  that  data  is  stored.  The  context  for  bookmarking  of  interactive  multimedia 
instruction  (IMI)  is  significantly  smaller  than  the  context  for  a  simulation. 


OTHER  APPLICABLE  EFFORTS 

There  are  several  other  efforts  under  the  auspices  of  SISO  that  potentially  have  bearing  on  this  effort. 

•  Military  Scenario  Definition  Language  (MSDL)  -  is  intended  to  provide  a  standard  mechanism 
for  loading  Military  Scenarios  independent  of  the  application  generating  or  using  the  scenario. 
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Standard  MSDL  is  defined  utilizing  an  XML  schema  thus  enabling  exchange  of  all  or  part  of 
scenarios  between  (e.g.)  Command  and  Control  (C2)  planning  applications,  simulations,  and 
scenario  development  applications.  Although  MSDL  is  only  applicable  to  military  training 
applications,  it  could  support  courseware  to  analyze  scenarios  to  dynamically  select  appropriate 
scenarios  or  to  dynamically  modify  or  create  a  scenario. 

•  Coalition  Battle  Management  Language  (C-BML)  is  an  unambiguous  language  used  to 
command  and  control  forces  and  equipment  conducting  military  operations,  and  to  provide  for 
situational  awareness  and  a  shared,  common  operational  picture.  C-BML  could  allow  courseware 
to  modify  or  create  behaviors  of  other  entities  such  as: 

•  threats  (e.g.  varying  shoot  orders  and  weapon  capabilities  to  adjust  difficulty;  or  even  adding 
or  disabling  anti-aircraft  artillery  (AAA)  or  Surface-to-Air  Missile  (SAM)  site), 

•  virtual  teammates  (e.g.  a  wingman),  and 

•  coaching  agents  (e.g.  "follow  me"). 

•  Like  MSDL,  C-BML  could  also  enable  courseware  to  analyze  the  behaviors  of  the  threats  in  the 
scenario  to  determine  appropriateness  to  the  student's  current  learning  need. 

•  The  Extensible  Modeling  and  Simulation  Framework  (XMSF)  is  defined  as  a  set  of  web-based 
technologies  and  services,  applied  within  an  extensible  framework,  that  enables  a  new  generation 
of  modeling  &  simulation  (M&S)  applications  to  emerge,  develop  and  interoperate.  Because 
XMSF  is,  by  definition,  web-based  like  SCORM,  there  is  a  ready  opportunity  to  integrate  such 
simulations  with  SCORM  with  fewer  of  the  interoperability  issues  associated  with  simulations 
designed  for  closed  or  stand-alone  environments  [20]. 
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Agenda 


ORGANIZATION 


•  The  SISO  SCORM-Sim  Study  Group 

-  SISO  Standards  Process 

-  Study  Group  Details 

•  SCORM-Sim  Integration  Challenges 

•  Study  Group  Scope  &  Products 

•  Use  Cases 

•  Taxonomy 

•  Architectures  &  Topologies 

•  Notional  Common  Architecture  &  Stds. 
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SCORM-Sim  Study  Group 

•  Joint  study  group  of  SISO  and  the  IEEE 
Learning  Technology  Standards  Committee 
(LTSC) 

•  Formed  to  study  standardization  for  the 
integration  of  simulation-based  learning 
experiences  with  Sharable  Content  Object 
Reference  Model  (SCORM)  environments 


irrr  Learning  Technology 
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ORGANIZATION 


“  SISO  Mission 

SISO  is  dedicated  to  the  promotion  of  modeling  and  simulation 
interoperability  and  reuse  through  the  exchange  of  ideas,  the 
examination  of  technologies,  and  the  development  of  standards. 


A  discussion  reflector  for  the  group  has  been  established  at: 

http://www.sisostds.org 

LTSC  Mission 

The  IEEE  Learning  Technology  Standards  Committee  (LTSC)  is 
chartered  by  the  IEE  Computer  Society  Standards  Activity  Board  to 
develop  accredited  technical  standards,  recommended  practices  and 
guides  for  learning  technology 

Additional  information  can  be  found  at: 
http://ieeeltsc.org/wg11CMI/cmi-sim 

srfks  ICCC  Learning  Technology 

ICCE  standards  Committee  fOJl  hOJ 
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Standards  Process 


ORGANIZATION 


•  Study  Group 

-  Created  as  the  result  of  a  Terms  of  Reference  (TOR) 

-  Chartered  to  answer  questions  of  interested 

-  Generally  have  a  limited  life  (-one  year) 

-  Can  be  the  first  step  to  a  Product  Development  Group  (PDG) 

•  Product  Development  Group 

-  Created  as  the  result  of  a  Product  Nomination  (PN) 

-  PDGs  perform  steps  2  &  3  of  the  BPDP 

-  PDG  Officers  and  the  SAC  perform  step  4  of  the  BPDP 

-  PDGs  exist  until  the  product  is  complete 

•  Product  Support  Group 

-  Established  once  the  PDG  has  completed  development 

•  Serve  as  the  Interpretations  Committee  for  IEEE  standards 

•  Responsible  for  steps  5  &  6  of  the  BPDP 
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Policies  &  Procedures 
SISO  SAC  Groups 


•  Study  Groups  (SGs) 

-  SISO  SGs  are  created  as  the  result  of  a  Terms  of  Reference  (TOR) 

-  Chartered  to  answer  questions  of  interest  to  the  M&S  community 

-  Have  a  limited  life;  generally  1  to  2  workshop  cycles 

-  Expected  to  lead  to  a  standards  activity 

-  Can  lead  to  a 

•  Standing  Study  Group  (SSG)  or 

•  Product  Development  Group  (PDG) 
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SCORM-Sim 

Study  Group 


ORGANIZATION 


•  Purpose 

-  Investigate  standard(s)  for  interfacing  Simulations  and  SCORM- 
Conformant  Learning  Content 

-  Enable  SCORM  content  or  Runtime  Environments  to  invoke  and 
communicate  with  simulations  in  a  standardized  and  interoperable 
fashion 

•  Products 

-  One  or  more  Product  Nominations  (PNs)  and/or  Product 
Authorization  Requests  (PARs)  based  on: 

•  Use  Cases 

•  Best  Practices  and  Design  Position  Papers 

•  Taxonomy  of  training  simulation  types 

•  Recommendations  on  stds  use  and/or  development 

•  Progress 

-  Collected  Position  Papers,  Uses  Cases,  Taxonomies 

-  Identified  Points  of  Interface 

-  Identified  existing  standards 

irrr  Learning  Technology 
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Organization 

•  Chair 

-  Brandt  W.  Dargue 

•  The  Boeing  Company,  St.  Louis,  MO,  USA 

•  Vice-Chair 

-  Geoffrey  Frank,  Ph.D. 

•  RTI  International,  Research  Triangle  Park  NC,  USA 

•  Secretary 

-  Brent  Smith 

•  Engineering  &  Computer  Simulations,  Orlando,  FL,  USA 

•  Technical  Area  Director 

-  Katherine  L.  Morse,  Ph.D. 

•  SAIC,  San  Diego,  CA,  USA 

•  Members 

-  Registered  on  the  PSG  reflector 

-  PSG  may  establish  additional  membership  requirements  with  approval 

-  PDG  may  allow  new  members  to  participate  in  business  pending  reflector 
registration 
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Other  Position  Paper 
Representatives 


•  Jack  Hyde,  AICC 

•  Avron  Barr,  Aldo  Ventures,  Inc. 

•  Bill  Ferguson,  BBN 

•  Robby  Robson,  Eduworks 

•  Luis  Arguello,  European  Space  Agency 

•  Bob  Pokorny,  Intelligent  Automation  Inc. 

•  Chris  Bray  &  Susan  Marshall,  Joint  ADL  Co-Lab 

•  Brian  Spaulding,  MAK  Technologies 

•  Michael  Freeman,  OSD 

•  Claude  Ostyn,  Ostyn  Consulting 

•  Shane  Gallagher,  SAIC 

•  Jim  Ong,  Stottler-Henke 

•  Carol  Wideman,  Vcom3D 
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ORGANIZATION 


Reflector  Members 


•  Aoyama,  Isao  (i-aoyama) 

•  Barr,  Avron  (Aldo) 

•  Blackmon,  Bill  (ADL) 

•  Bray,  Chris  (ADL) 

•  Bukai,  Ohad  (IAI) 

•  Conkey,  Curtis  (NAVAIR) 

•  Dargue,  Brandt  (Boeing) 

•  DuBuc,  Christopher 

•  Frank,  Geoffrey  (RTI) 

•  Furuichi,  Masakazu 

•  Gallagher,  Patrick 

•  Globe,  James  (L-3com) 

•  Haimson,  Craig 

•  Hughes,  Theresa 

•  Hyde,  Jack  (AICC) 

•  Lazarus,  Richard 

•  Lier,  Ad 
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Lowe,  Van  (UCF) 

Marshall,  Susan  (ADL) 

McGarity,  Michael  () 

Morissette,  Garry  (MAK) 

Myjak,  Michael  (Virtual  Workshop) 
Oehlert,  Mark  (BAH) 

Ostyn,  Claude 
Pittard,  Bryan 
Prescott,  Jonathan 
Robson,  Robby  (IEEE) 

Rouget,  Chris  (SISO) 

Smith,  Brent  (ECS) 
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Travers,  Virginia 
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SISO  Technical  Acceptance 

Principles 


Relevant 

-  SISO  Products  will  be  relevant  to  the  Modeling  and 
Simulation  Community. 

Substantive 

-  SISO  Products  shall  provide  meaningful  information  and/or 
results. 

Timely 

-  SISO  Products  will  be  produced  in  an  efficient  manner  to 
ensure  that  the  product  is  useful  to  the  community. 

Community  Review  &  Acceptance 

-  SISO  Products  will  be  reviewed  by  the  technical  community 
to  which  the  product  applies.  This  may  be  a  narrow  niche 
or  the  community  broadly  defined. 
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SISO  Balloted  Product 

Principles 


•  Generality 

-  Standards  Products  shall  be  as  general  as  possible,  while 
still  maintaining  usefulness,  to  support  the  broadest 
community  of  current  and  future  users. 

•  Stability 

-  Standards  Products  shall  be  established  and  changed  only 
as  necessary.  They  shall  be  prototyped  and  tested  before 
being  proposed  for  adoption  to  demonstrate  their  maturity. 

•  Supportability 

-  Standards  Products  shall  maintain  the  integrity  of  the 
existing  product  suite  and  the  needs  of  the  user. 


irrr  Learning  Technology 
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Challenges 


SCORM  -  Simulation  Interoperability  &  Run-Time  Issues 

•  How  does  a  training  simulation  fit  into  the  LMS/SCORM  paradigm? 

-  Non-Browser  delivery  environments 

•  Software  Distribution  to  Clients 

•  Network  Security  within  the  DoD 

-  Multiple  simultaneous  students  (scheduling,  lobby,  etc...) 

-  Two-way  runtime  communication  with  simulation 

•  Rolling  up  simulation  activity  into  assessment  measures 

•  Instructional  intervention  during  simulation 

-  Hints 

-  Coaching 

-  Intelligent  Tutoring 

•  After  Action  Review  is  an  essential  component  of  simulation-based  training 


Challenges  (Continued) 


•  According  to  SCORM  requirements,  all  communications  with  an  LMS  must  be 
done  through  a  standard  API  via  an  ECMAScript  (JavaScript) 

-  How  do  we  facilitate  communication  between  training  simulation  environment  (native 
code)  and  JavaScript  functions  in  LMS  web  page 

-  What  will  this  Communications  layer  will  handle: 

•  Initialization 

•  Translation  of  simulation  events  into  SCORM  global  objective  data  model  elements  (score, 
percent  successful,  percent  complete) 

•  Termination 

Assessment 

•  SCORM  specifies  a  simple  model  for  identifying  and  tracking  of  training 
objectives: 

-  Objective  ID 

-  Score  (if  applicable) 

-  Success  Status 

-  Completion  Status 

•  Training  simulation  assessment  data  must 
elements  in  order  for  a  SCORM  compliant 

irrr  Learning  Technology 
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Challenges  (Continued) 


What  is/are  the  architecture(s) 

-  Is  the  SCO  the  correct  integration  point 

State  saving  in  simulations 

-  Is  it  done  in  the  simulation  or  in  the  training  system? 

-  Can  it  be  used  to  “replay”  the  student’  activities  for  an  instructor? 

SCO  granularity 

-  Does  1  Simulation  =  1  SCO? 


Starting  up  and  shutting  down  (joining  and  leaving)  simulations 

-  Persistence 

-  Initializing  the  scenario 

Launching  and  maintaining  connectivity  with  multiple  assets  and/or  SCOs 
What  “counts”  as  a  simulation  in  the  instructional  context? 


-  “Taxonomy”  of  simulations 

•  Real  time  performance  assessment 

-  Where  and  how  is  assessment  done? 

-  Based  on  what  data  provided  by  whom? 

•  Individual  vs.  team  training 

-  Lobbying,  event  scheduling,  discovery,  coordination 

•  Security/policy 

-  What  types  of  assets  can  be  downloaded 

-  What  types  of  connections  are  allowed  (including  ports) 

•  Is  the  sharing/reuse  of  simulation  modules  in  or  out  of  scope? 

irrr  Learning  Technology 
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Terms  of  Reference  (TOR) 


•  Call  for  position  papers 


•  Survey  technical  and  pedagogical  approaches  taken 
to  date 

•  Determine  where  there  is  common  ground 

•  Discuss  potential  standardization  efforts 

•  Produce  appropriate  SISO  PNs  and/or  PARs  for 
IEEE  standards  projects 


< 


We  are  here! 


Proposed  Products 

1 .  Compilation  of  position  papers  to  develop 


1 .  Use  Cases 

2.  Common  Architectural  considerations 

3.  Identify  Key  Interface  Points 

2.  Identification  of  potential  standards 

3.  Final  Report  (the  intent  is  to  then  turn  this  into  a 
SISO  PDG  and/or  IEEE  Working  Group) 

IPPP  Learning  Technology 
ICCC  Standards  Committee 
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ORGANIZATION 


Use  Cases: 

•  Use  Case  Spreadsheet 

-  A  survey  to  identify  key  interface  points  and  supported  use  cases 
for  each  position  paper 

•  Joint  ADL  Co-Lab  developed  use  cases 

-  Modified  LSAL  template  use  cases 

-  “Lifecycle  of  learning”  use  cases 

•  AICC  Simulation  Topologies 

-  Communication  between  the  simulation  and  an  Instructional 
Component 

•  Generic  use  cases 

-  Courseware  uses  simulation 

-  Simulation  uses  courseware 
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Instructional  Use  Cases 


ORGANIZATION 


•  Courseware  (c/w)  Uses 
Simulation 

-  Show  me 

•  C/w  controls  (or  replays)  Sim  to 
demonstrate 

-  Teach  me 

•  Learner  controls  sim  for 
exploratory  learning 

-  Let  me  try 

•  Learner  controls  sim  for  practice 

-  Test  me 

•  Student  controls  sim  for 
assessment 

-  LMS*  initializes  and  receives 
scores  from  sim  as  activity 
node  in  course 


•  Simulation  Uses  Courseware 
As: 

-  Prerequisite  to  sim 

-  On-line  help/hints 

-  Remediation 

-  Sim  uses  LMS  *  to  store 
assessment  of  student  in  sim 
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Courseware  (c/w)  Uses  Simulation 


1 .  LMS  delivers  c/w  to  student. 

(a)  Prerequisite  training,  (b)  context/situation/scenario,  (c)  learning  objectives 

2.  C/w  initializes  simulation. 

(a)  Writes  out  1C*  file;  (b)  loads  sim  if  necessary;  (c)  establishes  interface; 

(d)  commands  sim  to  1C 

3.  Student  interacts  with  simulation. 

Options:  (a)  Student  performs  actions  in  sim; 

(b)  Student  observes  sim  “play  out  scenario;” 

4.  Simulation  sends  student  performance  data  to  c/w. 

Options  A:  (1)  Instantaneous  (real-time)  or  (2)  “after  action”  log 

Options  B:  (1)  Actions  or  results  of  actions  sent  to  c/w  to  assess;  (2)  scores  sent 

5.  C/w  sends  assessment  results  to  LMS. 

(a)  Immediately;  (b)  cached 

6.  LMS  selects  and  delivers  appropriate  content,  repeating  1-5  as  necessary. 

Options  A:  (1)  After  action  review;  (2)  remedial  c/w 
Options  B:  (1)  New  scenario;  (2)  scenario  adjustments 
Options  C:  (1)  Sim  stopped;  (2)  sim  kept  running; 
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A  Taxonomy  for 
Simulation-Based  Training 


ORGANIZATION 


User 

Multiplicity 

Instructor 

Location 

WRT 

Student 

Simulation 

Organization 

Simulation 

Location 

WRT 

Student 

Simulation 

Server 

Ownership 

Network 

Ownership 
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Local 

Centralized 

Local 
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Shared 
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Local 

Centralized 

Local 
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Local 
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